Context-based messaging system

ABSTRACT

Socially provided context-based messaging can include receiving a message content in a messaging system from a user, wherein the content includes objects; matching one or more objects of the message content to socially provided context information, which the socially provided context information is information relevant to an object obtained from a social network to which the user belongs; and displaying the socially provided context information with the message content for a recipient user to receive.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority from European PatentApplication 11178669.5, filed on Aug. 24, 2011, which is fullyincorporated herein by reference.

BACKGROUND

The background and description are provided in the environment ofmessaging systems used for augmentative and alternative communication(AAC) for communication for those with impairments or restrictions onthe production or comprehension of spoken or written language.

There are a number of different known systems to help users withdifferent impairments including picture-based systems in relation toinstant messaging or other messaging systems where the other user mayuse normal communication methods (typing/talking) or may also use AAC.There are existing AAC instant messaging applications in which a pictureboard is supplied adjacent to the chat window and the user generatestext messages by clicking the pictures. The received text may be spokento the end user.

While this works for high functioning AAC users, it is limited when itcomes to users who may have pragmatic language impairments (e.g.,Autism, Asperger's Syndrome, etc.). Often the context of a phrase may belost or the user may misinterpret the words. An AAC picture board maynot pick up on the user's idiosyncrasies.

Confusion may also happen if the other user is an AAC user as well.Selecting certain phrases on the AAC may have different contexts fordifferent users.

Although AAC pictures may come with textual context, without thiscontext pictures may be interpreted as having overlapping or similarmeanings. In some cases, users of the AAC board may interpret contextsbeyond the scope of recognized meanings leading to further confusion tothose they may communicate with.

BRIEF SUMMARY

According to a first aspect of the present invention there is provided amethod for socially provided context-based messaging, comprising:receiving a message content in a messaging system from a user, whereinthe content includes objects; matching one or more objects of themessage content to socially provided context information, wherein thesocially provided context information is information relevant to anobject obtained from a social network to which the user belongs; anddisplaying the socially provided context information with the messagecontent for a recipient user to receive.

According to a second aspect of the present invention there is provideda system for socially provided context-based messaging, comprising: adata processing system configured to execute a messaging clientapplication at which a message content is received from a user, whereinthe content includes objects; a matching system provided in a messagingsystem for matching one or more objects of the message content tosocially provided context information, wherein the socially providedcontext information is information relevant to an object obtained from asocial network to which the user belongs; and a receiving dataprocessing system configured to execute a messaging client applicationdisplaying the socially provided context information with the messagecontent for a recipient user to receive.

According to a third aspect of the present invention there is provided acomputer program product for socially provided context-based messaging.The computer program product includes a computer-readable storage mediumhaving stored thereon program code that, when executed, configures aprocessor to perform executable operations. The executable operationsinclude receiving a message content in a messaging system from a user,wherein the content includes objects; matching one or more objects ofthe message content to socially provided context information, whereinthe socially provided context information is information relevant to anobject obtained from a social network to which the user belongs; anddisplaying the socially provided context information with the messagecontent for a recipient user to receive.

According to a fourth aspect of the present invention there is provideda system. The system includes a processor configured to initiateexecutable operations. The executable operations include receiving amessage content in a messaging system from a user, wherein the contentincludes objects; matching one or more objects of the message content tosocially provided context information, wherein the socially providedcontext information is information relevant to an object obtained from asocial network to which the user belongs; and displaying the sociallyprovided context information with the message content for a recipientuser to receive.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Embodiments of the invention will now be described, by way of exampleonly, with reference to the accompanying drawings in which:

FIG. 1 is a block diagram of an example embodiment of a system inaccordance with a preferred embodiment of the present invention;

FIG. 2 is a block diagram of a computer system in which a preferredembodiment of the present invention may be implemented; and

FIG. 3 is a flow diagram of an example embodiment of a method inaccordance with a preferred embodiment of the present invention.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity. Further, where consideredappropriate, reference numbers may be repeated among the figures toindicate corresponding or analogous features.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding of embodiments of theinvention. However, it will be understood by those skilled in the artthat the embodiments of the present invention may be practiced withoutthese specific details. In other instances, well-known methods,procedures, and components have not been described in detail so as notto obscure the embodiments of the present invention.

One or more embodiments of the invention relate to the field ofmessaging systems. In particular, one or more embodiments of theinvention relate to socially provided context-based messaging systems.The described method and system provide messaging which providesrelevant socially-provided context information to either or both of themessaging participants using the content of a message input matched tosocially-provided context information from a social network of theparticipant.

In the environment of AAC, the described method and system may providecontext awareness in relation to the social environment to create a morerelevant set of images or other forms of information that would beunderstood by the AAC user. Existing AAC messaging systems may beimproved by taking a larger set of socially provided context informationinto account when determining additional information to display. In AACsystems, the social context may be moderated by a responsible thirdparty such as a care giver, therapist, or other moderator oradministrator. However, the described method and system may also beapplied to non-AAC applications to provide relevant socially providedcontext information, as described further below.

In the non-AAC environment, analysis of the social context (e.g. Web2.0)of one or both of the participants may be used to provide additionalrelevant social context information in the form of images or otherinformation. A non-AAC environment may use standard instant messaging orsocial networks (for example, Facebook (Facebook is a trade mark ofFacebook, Inc.), Google+ (Google+ is a trade mark of Google Inc.)). Thesystem may look for social contacts between the two people to help indefining a context (for example, “Bus stop” might refer to a day outthat one user would not recognize but another would).

The social context information may be provided in addition to theoriginal message content to provide more insight into the messagecontext for the receiving participant. The form of messaging system towhich the described method and system may be applied includes instantmessaging, email, web chat rooms or forums, short message service (SMS),and also messaging system using speech.

Semantic analysis of social context may be provided in order to provideadditional relevant information in messaging systems. Cognitiveconnections between the speaker and listener may be used. The system maydetermine the correct images or other information to display to both endusers (as part of a textual assist if needed). This can be used inoverlay of spoken language or through typed medium.

The system may use a combination of pictures, chat/voice and webcam/stored images on mobile device as needed to determine the socialcontext. The system also allows the user's social network to build trustcontexts with common images, to help others.

Referring to FIG. 1, a block diagram shows an example of the describedsystem 100. The system 100 may include three different types ofparticipants. A first type of participant may be the first user A 101who uses the described functionality of providing relevant socialcontext information based on a message input. In one embodiment, thefirst user A 101 may have communication impairments and may use AAC. Asecond type of participant may be a second user B 102, who may or maynot need to use AAC, in communication with the first user A 101. A thirdtype of participant may be a moderator M 103 of the first user A 101.

In the case in which the first user A 101 has communication impairmentsand uses AAC, the moderator M 103 may be a language therapist, a generalpractitioner, a caregiver, a social worker, etc. In the case in whichthe first user A 101 does not have communication impairments and doesnot use AAC, the moderator M 103 may be an administrator or the firstuser A 101 may moderate himself.

The first user A 101 and the second user B 102 and, optionally, themoderator M 103, may have messaging client applications 111, 112, 113which include graphical user interfaces (GUI) which may have additionalfunctionality added for displaying relevant social context informationfor a message input.

The system 100 may include a messaging server 120 which supports theadditional functionality of providing relevant social contextinformation for a message input and may include or have access to amatching system 130. The matching system 130 may provide a mappingbetween objects 131 of a message input which may be in the form ofwords, phrases, images, etc. and context information 132 which may be inthe form of images, data, text, etc. The matching system 130 may providerelationship meta-data 133 between objects 131 and context information132.

Participant context information 141 may be stored and collected fromdatabases or devices 140 of the user A 101. For example, such databasesor devices may include personal data, images, etc.

A social network 150 may be provided of contributors of social contextinformation 151 for user A 101. The social network 150 may be anexisting social network to which user A 101 belongs. The social network150 may have different settings allowing different levels ofcontribution by contributors, for example, with the user A 101 or, inthe case of a user A 101 requiring AAC, a moderator M 103 having thehighest access. A plugin engine may be provided to interface with asocial network 150 to act as a bridge between the required functionalityof the described system of obtaining context information 151 and withsocial network APIs (for example, OpenSocial API (OpenSocial is a trademark of Google Inc.) or Facebook API).

The moderator M 103 is particularly applicable in the embodiment inwhich user A 101 requires AAC. The moderator M 103 can oversee thesocial context information on behalf of user A 101. The moderator M 103may have a moderator system 160 for provision of additional contextinformation 161 for a user. The moderator system 160 may include ashadow information component 162.

The matching system 130 may include an information receiving component134 for receiving context information 141 from the participant databasesand devices 140, social context information 151 from the social network150, and/or moderator's context information from a moderator's externalsite 160. The various forms of context information 141, 151, 161 may bestored as context information 132 matched to an object 131 withrelationship meta-data 133.

A linguistic engine 170 may be provided at a messaging client 111,messaging server 120, or matching engine 130, for breaking down a textor audio input and recognizing objects 131 within the message input formatching to context information 132.

The matching system 130 may also include a searching component 190 forsearching the related objects 131 and context information 132. A filtercomponent 191 may be provided for filtering the relationship meta-data133.

A moderator M 103 may provide shadow information 135 which is contextinformation which the moderator M 103 determines is more appropriate toan object 131 than given context information 132. For example, in theembodiment of a user A 101 who requires AAC, the moderator M 103 may tryto reinforce more appropriate context information relating to an object131 instead of the context information 132 currently associated with theobject 131.

The matching system 130 may include additional comments 136 associatedwith context information 132. A related comments system 180 may beprovided, as part of the matching system 130 or accessible by thematching system 130, allowing contributors or participants in messagingwith a user A 101 to view the additional comments 136 on the contextinformation 132 provided corresponding to an object 131. The relatedcomments system 180 relates to optional features presented toparticipants given they have the correct permissions to view/edit.Example features include:

-   -   Read a comment.    -   Edit a comment.    -   Vote on a comment attached to an image.    -   Moderate comments (approve/delete/edit).    -   A list of related people in the social network of user A with        the option to contact them. If this is used then the person        contacted may get a breakdown of related images/conversations        they are being contacted with regard to (this can be        configured).    -   See a list of other common images that may have a better context        to that image.    -   See a paired image that user A will see but which is not        correct.    -   If the image is a shadow image, and if so at what level should        it be visible.

Referring to FIG. 2, an exemplary system for implementing aspects of theinvention includes a data processing system 200 suitable for storingand/or executing program code including at least one processor 201coupled directly or indirectly to memory elements through a bus system203. The memory elements can include local memory employed during actualexecution of the program code, bulk storage, and cache memories whichprovide temporary storage of at least some program code in order toreduce the number of times code must be retrieved from bulk storageduring execution.

The memory elements may include system memory 202 in the form of readonly memory (ROM) 204 and random access memory (RAM) 205. A basicinput/output system (BIOS) 206 may be stored in ROM 204. System software207 may be stored in RAM 205 including operating system software 208.Software applications 210 may also be stored in RAM 205.

The system 200 may also include a primary storage means 211 such as amagnetic hard disk drive and secondary storage means 212 such as amagnetic disc drive and an optical disc drive. The drives and theirassociated computer-readable media provide non-volatile storage ofcomputer-executable instructions, data structures, program modules andother data for the system 200. Software applications may be stored onthe primary and secondary storage means 211, 212 as well as the systemmemory 202.

The computing system 200 may operate in a networked environment usinglogical connections to one or more remote computers via a networkadapter 216.

Input/output devices 213 can be coupled to the system either directly orthrough intervening I/O controllers. A user may enter commands andinformation into the system 200 through input devices such as akeyboard, pointing device, or other input devices (for example,microphone, joy stick, game pad, satellite dish, scanner, or the like).Output devices may include speakers, printers, etc. A display device 214is also connected to system bus 203 via an interface, such as videoadapter 215.

Referring to FIG. 3, a flow diagram shows an embodiment of the method300 carried out by the matching system. A message content as input by auser is received 301 in the messaging system. The content may be brokendown 302 by a linguistic engine into its related objects. The relatedobjects may be words, phrases, numbers, of the content as well asimages, attachments, etc.

Existing stored context information may be searched 303 to match theobjects. Social context information may be obtained 304 from one or moresocial networks of the participant providing the message input. Otheradditional context information may be obtained by being pulled from auser's database of images/devices, and/or may be obtained from anexternal site provided by a moderator.

The relationship of the context information to an object may begenerated 305 with meta-data, for example, including the following:

-   -   Text comment of image.    -   Social comment (ranked on votes).    -   Relationship of entity to the person being conversed with.    -   Global Positioning System (GPS) data.    -   Image recognition. E.g. Face detection.    -   Other Exchangeable Image File Format (EXIF) information of        image.    -   Moderator flagged images for that object.    -   Relationship of entity to another entity within the sentence.        E.g. “[Ann]” can mean Mother or cousin depending on context.    -   Date/Time relationship. E.g. “[ZOO] [LAST SATURDAY]” vs. “[ZOO]        [LAST YEAR]”.    -   Images flagged by the moderator.    -   Stored audio on device. Example would be a picture taken and        audio at that time flagging to match to the entity through        voice/sound recognition.

A list of relevant context information (for example, images and data)may be generated 306 against the objects of the message content input.

It may be determined 307 if the list is empty. If so, an object is notmatched and an “Unknown” image may be displayed 308 (for example, “?”).The person who input the text with the object may be flagged 309 tocorrect the missing information. The object may also be flagged for theattention of a moderator. Optionally, the user may be passed to anexternal site to find matching context information, such as an image.

If the list is not empty, each of the context information elementswithin the generated list may be checked for any shadow information orrelated comments information. For example, shadow information may beprovided where a participant is using the incorrect image and amoderator has selected another to take its place. Related commentsinformation may be provided via a social network to provide insight intothe context information.

The list may then be filtered to display 310 the most likely contextinformation. The precedence of meta-data may be configured by a user ormoderator to set the relevance of meta-data entries.

In the event of an element of context information having shadowinformation, then based on the data, the shadow information may beoverlaid 311 to slowly force a user to accept the correct information.Examples of methods of overlays can be opacity, grouped together, orappear on mouse over.

In the event of an element of context information having relatedcomments information, and the receiving participant having theauthorization to view the comments, these comments may also be displayed312.

In one embodiment of a messaging system in the form of an AAC enabledinstant messaging system, the system may be initially configured withthe following settings.

1. A standard picture cards layout for an instant messaging window withpicture cards corresponding to or mapped to text input such as words orphrases.

2. Pre-defined images for the user. (For example, common items that theuser uses.)

3. Security settings for social network users. Sample settings include:

-   -   Read/Write comments to existing pictures.    -   Moderate comments.    -   Vote on comments (agree/disagree).    -   Allow others to contact those within the social network.    -   Ability to generate reports.

4. The user devices may be queried either live or via sync results. Someexamples of devices may include:

-   -   A personal digital assistant (PDA)/AAC device.    -   Internet sites. (for example, social network sites, image        hosting services, etc.).    -   Machines/Shared drives in relation to a social network.

The following are example conversational workflows in an instantmessaging system with a user A who requires AAC and in which the contextinformation is provided in the form of images.

Actors:

-   -   User (A)—User of the system. Person is language impaired.    -   User (B)—Normal user. Does not have the system installed.    -   User (C)—Another user of the system. Person may or may not be        language impaired.    -   User (M)—Moderator (e.g. LT/GP/caregiver/social worker)

(B)→(A) interaction.

1. User opens an instant message window. The user is warned that theperson they are communicating with is using the system. It may ask themto install the related system, or refer them to an interface via anothersystem (e.g. Web site).

2. (B) writes in their message to (A).

3. The system takes the message and attempts to break it down intorelated sentence objects (e.g. Subject, Object, Verb). This is done bylinguistic analysis.

4. Each object is mapped to a related image in the system.

5. The image is displayed to (A).

6. Optionally, the image can also be displayed to (B) with mouse overtext which gives details as to the sentence object and other notes as towhy it was picked.

7. Also optionally, the system may URL hot link the image to point to aconnection to the system to give more in depth details.

(A)→(B) interaction.

1. (A) opens a chat window to (B) or is responding to existing message.

2. (A) may select from a standard picture board or type text. Within thepicture board is the picture tree to allow them to map to particulardevices. If they are typing text, then related images may pop up in thesystem for them to quickly select from if needed.

3. Once the message is complete, the images are sent to (B) to read. (B)may also receive a text based description of the pictures.

4. Depending on settings (B) may be able to see text on mouse over orURL links to better details (as per B→A interaction).

(C)→(A) interaction.

1. The user opens a chat window or is responding to an incoming message.

2. As the user types, the related sentence object may be highlighted andrelated images may be shown.

3. The user can click on a related image that best describes what it isthey are trying to say. When they do this, IF they have the rights theymay see related comments by others.

4. With text selected against an image, the user can if they have therights to do so, map that image to their text, as well as add additionalcomments to the image. This may be tagged with the author of thecomments, date and time. There may also be a voting option forconfidence of the related image.

5. Where no match is found at all, then a generic “Don't know” image maybe used. Alternatively they may opt to call someone within (A)'s socialnetwork to get a better image.

6. The user sends the message to (A).

7. Optionally, where language breaks down (C) may flag thelines/conversation for review by a moderator.

(M) interaction.

The moderator may get a breakdown of common images used, their contextand interaction with users. They can apply rights to other users, andmodify full settings of the system. The moderator system may be fullytiered for different users. As the system priority is the language of(A), modifying images within (A)'s picture board tree or context may beset to a highest security setting. The moderator may also be able to seeoverlaid (shadowed) images (from C→interaction).

The following is an example of a scenario of an AAC user conversation.

Conversation:

-   -   Person B: “Did you enjoy the Zoo last Saturday?”    -   Person A: “Soda”

In this case it appears meaningless unless Person B is familiar with thecontext of the word “Soda”.

This response does not make much sense. Does Person A want a drink? Didsomething happen in relation to the zoo+soda? Is Person A actually inthe conversation?

With the described system, the context of the words/pictures may bemapped to existing incidents recorded in the system. This may be throughother conversations, what is pre-programmed or associated with storedrecordings (e.g. GPS to image mapped data), as well as social baseddescriptive comments.

So in the example above, the following context is provided:

Person B

-   -   [Zoo] [Saturday]—Application scans the EXIF image information        that maps to that date and GPS information which would flag a        relationship with Saturday and a Zoo recently.    -   [Fun]—Looks for images which may be flagged as positive key good        events.

Person B is given a list of images from which to select to get theircontext understood. The response is more positive now.

Going back to the initial response of “Soda”, the system refers back tocommon instances of “Soda” used previously in the system. It also checksfor comments attached to the word which others users have added toexplain the context. In this case it may come back with:

-   -   “I don't understand”*    -   “Confusion”    -   “Accident”    -   “Drink”    -   “Thirsty”

* With a tagged comment of: “Spilt soda when younger because he gotconfused on what I asked. Equates the word “soda” with confusion or notunderstanding”—Anne (Mother) Jan. 5, 2011.

This comment may further be voted up/down by others within the socialnetwork to help reinforce known contexts. The explanations may also belocked down to known instant message contacts. Users within the socialnetwork may also opt to have conversations tagged to troublesome wordsto help others guess what may be described.

In other instances the system may translate for the person at the otherend. So if they were to say “I don't understand” the correct context isshown to Person A but is also shown the “Soda” image to help link theresponse. This can slowly fade out the “Soda” image until it is nolonger visible once the correct term is determined to attempt to avoidincorrect reinforcers.

The data collected may also be reviewed at a later point reviewed by aLanguage Therapist (LT)/GP/caregiver/social worker to help improve theuser's quality of life.

The embodiments of the invention can take the form of an entirelyhardware embodiment or an embodiment containing both hardware andsoftware elements. In a preferred embodiment, the invention isimplemented in software within suitable hardware, which includes but isnot limited to firmware, resident software, microcode, etc.

The invention can take the form of a computer program product accessiblefrom a computer-usable or computer-readable medium providing programcode for use by or in connection with a computer or any instructionexecution system. For the purposes of this description, a computerusable or computer readable medium can be any apparatus that cancontain, store, communicate, propagate, or transport the program for useby or in connection with the instruction execution system, apparatus ordevice.

The medium can be an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system (or apparatus or device) or apropagation medium. Examples of a computer-readable medium include asemiconductor or solid state memory, magnetic tape, a removable computerdiskette, a random access memory (RAM), a read only memory (ROM), arigid magnetic disk and an optical disk. Current examples of opticaldisks include compact disk read only memory (CD-ROM), compact diskread/write (CD-R/W), and DVD.

Improvements and modifications can be made to the foregoing withoutdeparting from the scope of the present invention.

What is claimed is:
 1. A computer-implemented method within a matchingcomputer hardware system, comprising: receiving, from a sender, messagecontent; breaking down the content into at least one object; matchingthe at least one object with previously-stored, socially-providedcontext information; providing, to an augmentative and alternativecommunication (AAC)-enabled client application of a user, the messagecontent and the socially-provided content information, wherein theAAC-enabled client application is configured to display the messagecontent along with the socially-provided context information, and thesocially-provided context information is information relevant to anobject within the content and provided by members of a social network towhich the sender providing the message content belongs.
 2. The method ofclaim 1, wherein relationship metadata links the object to thesocially-provided context information.
 3. The method of claim 1, whereinadditional context information is matched to the object within themessage content, and the additional context information is obtained froma user's device.
 4. The method of claim 1, wherein the socially-providedcontext information is stored in relation to the object for futureaccess.
 5. The method of claim 1, wherein the socially provided contextinformation is moderated.
 6. The method of claim 1, wherein thesocially-provided context information provides insight of an AAC user'sunderstanding of the object in the message content.
 7. A matchingcomputer hardware system, comprising: a hardware processor configured toinitiate the following executable operations: receiving, from a sender,message content; breaking down the content into at least one object;matching the at least one object with previously-stored,socially-provided context information; providing, to an augmentative andalternative communication (AAC)-enabled client application of a user,the message content and the socially-provided content information,wherein the AAC-enabled client application is configured to display themessage content along with the socially-provided context information,and the socially-provided context information is information relevant toan object within the content and provided by members of a social networkto which the sender providing the message content belongs.
 8. The systemof claim 7, wherein relationship metadata links the object to thesocially-provided context information.
 9. The system of claim 7, whereinadditional context information is matched to the object within themessage content, and the additional context information is obtained froma user's device.
 10. The system of claim 7, wherein thesocially-provided context information is stored in relation to theobject for future access.
 11. The system of claim 7, wherein thesocially provided context information is moderated.
 12. The system ofclaim 7, wherein the socially-provided context information providesinsight of an AAC user's understanding of the object in the messagecontent.
 13. A computer program product, comprising: a hardware storagedevice having computer usable program code stored therein, the computerusable program code, which when executed by a matching computer hardwaresystem, causes the matching computer hardware system to perform:receiving, from a sender, message content; breaking down the contentinto at least one object; matching the at least one object withpreviously-stored, socially-provided context information; providing, toan augmentative and alternative communication (AAC)-enabled clientapplication of a user, the message content and the socially-providedcontent information, wherein the AAC-enabled client application isconfigured to display the message content along with thesocially-provided context information, and the socially-provided contextinformation is information relevant to an object within the content andprovided by members of a social network to which the sender providingthe message content belongs.
 14. The computer program product of claim13, wherein relationship metadata links the object to thesocially-provided context information.
 15. The computer program productof claim 13, wherein additional context information is matched to theobject within the message content, and the additional contextinformation is obtained from a user's device.
 16. The computer programproduct of claim 13, wherein the socially-provided context informationis stored in relation to the object for future access.
 17. The computerprogram product of claim 13, wherein the socially provided contextinformation is moderated.
 18. The computer program product of claim 13,wherein the socially-provided context information provides insight of anAAC user's understanding of the object in the message content.